Advanced Intelligent Systems

Deep learning-based virtual fiber staining provides a promising complement to routine H&E pathology. However, the reliance on predefined staining style inputs and manual intervention limits the clinical applicability of existing methods. To address these challenges, we introduce ViFIT-assisted histopathology, a two-stage diagnostic approach that integrates our proposed unsupervised deep learning-based virtual fiber transformation model (ViFIT). This approach enables the conversion of H&E-stained images with diverse styles into pathologist-preferred H&E images, while simultaneously generating content-consistent virtual fiber images containing label-free collagen fibers and stained reticular and elastic fibers. ViFIT-assisted histopathology reveals tumor-associated fibers and provides quantitative metrics across multiple intraoperative and postoperative cases. Experimental results demonstrate that ViFIT significantly outperforms state-of-the-art unsupervised methods in both style standardization and virtual staining, across various downstream tasks and cancer types. By eliminating the need for staining variation and manual annotation, ViFIT-assisted histopathology streamlines histopathology workflows, making it well-suited for multi-center consultations and differential diagnosis.

Cytopathology plays a central role in the early detection of cancers such as cervical, lung, and bladder cancer due to its speed, simplicity, and minimally invasive nature. However, its effectiveness is limited by variability in diagnostic accuracy stemming from subjective visual interpretation. Although many AI-powered systems have been proposed to improve consistency, none have achieved fully autonomous, clinical-grade performance. Existing approaches serve as assistive tools and still rely on human oversight for interpretation and decision-making. Here we present a clinical-grade autonomous cytopathology pipeline that combines high-resolution, real-time optical whole-slide tomography with edge computing to deliver end-to-end automation. The system achieves practical performance in imaging speed, quality, and data volume, with localized data compression enabling streamlined storage and accelerated AI-driven analysis. In addition to supporting cell-level classification, the platform enables flow cytometry-like, population-wide morphological profiling for comprehensive interpretation of cellular distributions and patterns. A vision transformer achieved area-under-receiver-operating-characteristic-curve values exceeding 0.99 for detecting LSIL, HSIL, and adenocarcinoma. In a clinical cohort of cervical liquid-based cytology samples from 318 donors, LSIL counts strongly correlated with HPV positivity, while HSIL counts scaled with diagnostic severity. The system enables autonomous triage cytology, offering a foundation for routine, scalable, and objective diagnostics.

Multiplex staining enables simultaneous detection of multiple protein markers within a tissue sample. However, the increased marker count increased the likelihood of staining and imaging failure, leading to higher resource usage in multiplex staining and imaging. We address this by proposing a deep learning-based MArker imputation model for multipleX IMages (MAXIM) that accurately impute protein markers by leveraging latent biological relationships between markers. The models imputation ability is extensively evaluated at pixel and cell levels across various cancer types. Additionally, we present a comparison between imputed and actual marker images within the context of a downstream cell classification task. The MAXIM models interpretability is enhanced by gaining insights into the contribution of individual markers in the imputation process. In practice, MAXIM can reduce the cost and time of multiplex staining and image acquisition by accurately imputing protein markers affected by staining issues.

Interest is growing rapidly in using deep learning to classify biomedical images, and interpreting these deep-learned models is necessary for life-critical decisions and scientific discovery. Effective interpretation techniques accelerate biomarker discovery and provide new insights into the etiology, diagnosis, and treatment of disease. Most interpretation techniques aim to discover spatially-salient regions within images, but few techniques consider imagery with multiple channels of information. For instance, highly multiplexed tumor and tissue images have 30-100 channels and require interpretation methods that work across many channels to provide deep molecular insights. We propose a novel channel embedding method that extracts features from each channel. We then use these features to train a classifier for prediction. Using this channel embedding, we apply an interpretation method to rank the most discriminative channels. To validate our approach, we conduct an ablation study on a synthetic dataset. Moreover, we demonstrate that our method aligns with biological findings on highly multiplexed images of breast cancer cells while outperforming baseline pipelines.

Vision-language models (VLMs) provide a unified framework for multimodal reasoning, yet their representations are primarily learned from natural image-text corpora and often exhibit semantic misalignment when transferred to histopathology, particularly under data-limited diagnostic settings. To address this limitation, we propose HistoSB-Net, a semantic bridging network designed to adapt pre-trained VLMs to multimodal histopathological diagnosis while preserving their original semantic structure. HistoSB-Net introduces a constrained semantic bridging (CSB) module that operates within the self-attention projection space of both vision and text encoders. Instead of employing explicit cross-attention or full fine-tuning, CSB adaptively modulates pre-trained attention projections through a lightweight nonlinear semantic bottleneck, enabling structured cross-modal regulation with limited additional parameters. The framework supports both patch-level and whole-slide image (WSI)-level diagnosis within a unified architecture. Experiments on six pathology benchmarks, comprising two WSI-level and four patch-level datasets, demonstrate consistent improvements over zero-shot inference across 36 backbone-dataset combinations under limited supervision. Further analysis of prototype-based margin distributions and confusion matrices shows that these improvements are accompanied by enhanced intra-class compactness and increased inter-class separation in the embedding space. These results indicate that CSB provides an effective and computationally manageable strategy for adapting pre-trained VLMs to data-limited digital pathology tasks.

Three-dimensional (3D) histopathology involves the microscopic examination of a specimen, which plays a vital role in studying tissues 3D structures and the signs of diseases. However, acquiring high-quality histological images of a whole organ is extremely time-consuming (e.g., several weeks) and laborious, as the organ has to be sectioned into hundreds or thousands of slices for imaging. Besides, the acquired images are required to undergo a complicated image registration process for 3D reconstruction. Here, by incorporating a recently developed vibratome-assisted block-face imaging technique with deep learning, we developed a pipeline termed HistoTRUST that can rapidly and automatically generate subcellular whole organs virtual hematoxylin and eosin (H&E) stained histological images which can be reconstructed into 3D by simple image stacking (i.e., without registration). The performance and robustness of HistoTRUST have been successfully validated by imaging all vital mouse organs (brain, liver, kidney, heart, lung, and spleen) within 1-3 days depending on the size. The generated 3D dataset has the same color tune as the traditional H&E stained histological images. Therefore, the virtual H&E stained images can be directly analyzed by pathologists. HistoTRUST has a high potential to serve as a new standard in providing 3D histology for research or clinical applications.

With the fact that protein functionality is tied to its structure and shape, a protein design paradigm of generating proteins tailored to specific shape contexts has been utilized for various biological applications. Recently, researchers have shown that top-down strategies are possible with the aid of deep learning for the shape-conditioned design. However, state-of-the-art models have limitations because they do not fully consider the geometric and chemical constraints of the entire shape. In response, we propose ShapeProt, a pioneering end-to-end protein design framework that directly generates protein surfaces and generate sequences with considering the entire nature of the generated shapes. ShapeProt distinguishes itself from current protein deep learning models that primarily handle sequence or structure data because ShapeProt directly handles surfaces. ShapeProt framework employs mask-based inpainting and conditioning to generate diverse shapes at the desired location, and these shapes are then translated into sequences using a shape-conditioned language model. Drawing upon various experimental results, we first prove the feasibility of generative design directly on the three-dimensional molecular surfaces beyond sequences and structures.

Whole slide image (WSI) classification plays a crucial role in digital pathology data analysis. However, the immense size of WSIs and the absence of fine-grained sub-region labels, such as patches, pose significant challenges for accurate WSI classification. Typical classification-driven deep learning methods often struggle to generate compact image representations, which can compromise the robustness of WSI classification. In this study, we address this challenge by incorporating both discriminative and contrastive learning techniques for WSI classification. Different from the extant contrastive learning methods for WSI classification that primarily assign pseudo labels to patches based on the WSI-level labels, our approach takes a different route to directly focus on constructing positive and negative samples at the WSI-level. Specifically, we select a subset of representative and informative patches to represent WSIs and create positive and negative samples at the WSI-level, allowing us to better capture WSI-level information and increase the likelihood of effectively learning informative features. Experimental results on two datasets and ablation studies have demonstrated that our method significantly improved the WSI classification performance compared to state-of-the-art deep learning methods and enabled learning of informative features that promoted robustness of the WSI classification.

Histological haematoxylin and eosin-stained (H&E) tissue sections are used as the gold standard for pathologic detection of cancer, tumour margin detection, and disease diagnosis1. Producing H&E sections, however, is invasive and time-consuming. Non-invasive optical imaging modalities, such as optical coherence tomography (OCT), permit label-free, micron-scale 3D imaging of biological tissue microstructure with significant depth (up to 1mm) and large fields-of-view2, but are difficult to interpret and correlate with clinical ground truth without specialized training3. Here we introduce the concept of a virtual biopsy, using generative neural networks to synthesize virtual H&E sections from OCT images. To do so we have developed a novel technique, "optical barcoding", which has allowed us to repeatedly extract the 2D OCT slice from a 3D OCT volume that corresponds to a given H&E tissue section, with very high alignment precision down to 25 microns. Using 1,005 prospectively collected human skin sections from Mohs surgery operations of 71 patients, we constructed the largest dataset of H&E images and their corresponding precisely aligned OCT images, and trained a conditional generative adversarial network4 on these image pairs. Our results demonstrate the ability to use OCT images to generate high-fidelity virtual H&E sections and entire 3D H&E volumes. Applying this trained neural network to in vivo OCT images should enable physicians to readily incorporate OCT imaging into their clinical practice, reducing the number of unnecessary biopsy procedures.

AO_SCPLOWBSTRACTC_SCPLOWO_ST_ABSImportanceC_ST_ABSManual annotation of histopathological images by pathologists is effort-intensive and represents a major challenge for computational analysis and clinical AI deployment. ObjectiveTo develop and validate an open-source semi-supervised deep learning framework for accurate and efficient annotation and analysis of cellular structures in whole slide images. DesignMethods development study using active contrastive learning with iterative human-in-the-loop feedback. The framework integrates cell segmentation, embedding-based visualization, and semi-supervised classification. SettingDigital pathology computational analysis platform with intuitive web-based annotation interface. MethodsFive whole slide images of canine invasive urothelial carcinoma digitized at 40x magnification, representing low, intermediate, and high histological grades with diverse morphological patterns. The AnnotateAnyCell framework processing cells through Cell-pose segmentation, UMAP dimensionality reduction for interactive annotation, and contrastive learning-based classification with pseudolabel generation. Main Outcomes and MeasuresPrimary outcomes were accurate classification for nuclear morphology features (mitotic figures, nucleoli, chromatin, shape) and annotation efficiency measured as time per cell. Secondary outcomes included inter-annotator agreement and model performance scaling with training data size. ResultsThe framework processed 8 slides containing hundreds of thousands of cells. Latent space clustering-based annotation required 47 minutes versus 63 minutes for sequential annotation (25% reduction, 95% CI 18%-32%). Classification accuracy reached 96.3% {+/-} 1.2% for mitotic figures and 98.3% {+/-} 1.4% for nucleoli with 1075 labeled samples. Nuclear shape classification achieved 59.5% {+/-} 2.1% accuracy. Inter-annotator agreement was highest for chromatin (100%) and nucleoli (95%,{kappa} = 0.95), moderate for mitotic figures (64%,{kappa} = 0.58), and lowest for shape (81%,{kappa} = 0.36). Performance scaled efficiently, with nucleoli classification reaching 95.5% {+/-} 1.5% accuracy using only 215 samples. Conclusions and RelevanceThis semi-supervised active learning framework substantially reduces annotation burden while achieving expert-level accuracy for well-defined morphological features. The open-source tool accelerates histopathological dataset curation for cancer research and enables deployment of AI-assisted diagnostics in resource-constrained settings. Githubhttps://github.com/shouryaverma/AnnotateAnyCell Key PointsO_ST_ABSQuestionC_ST_ABSCan a semi-supervised deep learning framework reduce annotation burden for pathologist while maintaining accuracy in digital pathology? FindingsIn this methods study evaluating eight canine urothelial carcinoma slides, the AnnotateAnyCell framework achieved 96% accuracy for mitotic classification and 98% for nucleoli classification using just 1075 labeled samples, while reducing annotation time by 25% compared to sequential annotation. MeaningThis open-source framework enables efficient large-scale cellular annotation in histopathology, potentially accelerating cancer research and clinical application.

Histologically stained tissue sections are considered the gold standard for studying microscopic anatomy and diagnosing disease in clinical practice. However, the processes of sectioning and staining are laborious, and the overall method relies on two-dimensional (2D) analysis. In contrast, X-ray-based virtual histology offers the advantage of virtual sectioning while retaining the full three-dimensional (3D) volumetric representation of the tissue. Nevertheless, its grayscale nature has prevented it to be readily utilized by pathologists who are accustomed to conventional histological stains. In this work, we present a histology-guided enhancement platform that can integrate the 3D information provided by synchrotron radiation phase-contrast microCT with the rich visual features characteristic of histological stains. We introduce a multi-stage microCT-histology co-registration method combined with a virtual staining deep neural network and demonstrate successful virtual histological staining of microCT human and mouse lung tissue that closely resembles standard histology. We evaluate our strategy on multiple histological stains and apply it to identify 3D collagen-based remodeling of pulmonary arteries in patients with pulmonary hypertension. Overall, this innovative enhancement pipeline has the potential to aid in the incorporation of microCT into clinical practice, and advance non-destructive 3D pathology for improved diagnostic efficiency and accuracy.

AbstractHuman pluripotent stem cell (hPSC)-derived cardiac organoid is the most recent three-dimensional tissue structure that mimics the structure and functionality of the human heart and plays a pivotal role in modeling heart development and disease. The hPSC-derived cardiac organoids are commonly characterized by bright-field microscopic imaging for tracking daily organoid differentiation and morphology formation. Although the brightfield microscope provides essential information about hPSC- derived cardiac organoids, such as morphology, size, and general structure, it does not extend our understanding of cardiac organoids on cell type-specific distribution and structure. Then, fluorescence microscopic imaging is required to identify the specific cardiovascular cell types in the hPSC-derived cardiac organoids by fluorescence immunostaining fixed organoid samples or fluorescence reporter imaging of live organoids. Both approaches require extra steps of experiments and techniques and do not provide general information on hPSC-derived cardiac organoids from different batches of differentiation and characterization, which limits the biomedical applications of hPSC-derived cardiac organoids. This research addresses this limitation by proposing a comprehensive workflow for colorizing phase contrast images of cardiac organoids from brightfield microscopic imaging using conditional Generative Adversarial Networks (GANs) to provide cardiovascular cell type-specific information in hPSC-derived cardiac organoids. By infusing these phase contrast images with accurate fluorescence colorization, our approach aims to unlock the hidden wealth of cell type, structure, and further quantifications of fluorescence intensity and area, for better characterizing hPSC-derived cardiac organoids.

Quantitative characterizations of cellular dynamics and features of individual cells from a large heterogenous population is essential to identify rare, disease-driving cells, which often exhibit aberrant cellular behaviors like abnormal division, aggressive migration or irregular phylogenetic cell lineages. A recent development in the combination of high-throughput screening microscopy with single cell profiling provides an unprecedented opportunity to decipher the underlying mechanisms of disease-driving phenotypes observed under a microscope. However, accurately and instantly processing large amounts of image data like longitudinal time lapse movies remains a technical challenge when an immediate analysis output (in minutes) of quantitative characterizations is required after data acquisition. Here we present a Fast and Accurate real-time Cell Tracking (FACT) algorithm, which combines GPU-based, ground truth-assisted trainable Weka segmentation and real-time Gaussian mixture model-based cell linking. FACT also implements an automatic cell track correction function to improve the tracking accuracy. With FACT, we can segment [~]20,000 cells in 2 seconds ([~]4.5-27.5 times faster than state-of-the-art), and can export quantifiable features from the cell tracking results minutes after data acquisition (independent of the number of acquired image frames) with average 90-95% tracking precision. Such performance is not feasible with state-of-the-art cell tracking algorithms. We applied FACT to real-time identify directionally migrating glioblastoma cells with 96% precision and to identify rare, irregular cell lineages in a population of [~]10,000 cells from a 24hr-time lapse movie with an average 91% F1 score, results from both were exported instantly, mere minutes after image acquisition.

Histopathology is considered the reference standard for diagnosing the presence and nature of many malignancies, including cancer. However, analyzing tissue samples and writing pathology reports is time-consuming, labor-intensive, and non-standardized. To address this problem, we present HistoGPT, the first vision language model that simultaneously generates reports from multiple pathology images. It was trained on more than 15,000 whole slide images from over 6,000 dermatology patients with corresponding pathology reports. The generated reports match the quality of human-written reports, as confirmed by a variety of natural language processing metrics and domain expert evaluations. We show that HistoGPT generalizes to six geographically diverse cohorts and can predict tumor subtypes and tumor thickness in a zero-shot fashion. Our model demonstrates the potential of an AI assistant that supports pathologists in evaluating, reporting, and understanding routine dermatopathology cases.

Advances in high-throughput imaging and experimental automation have dramatically increased the scale of biological datasets, creating a growing need for tools that can efficiently identify and localize features in complex image data. Although deep learning has transformed image analysis, methods such as region-based convolutional neural networks remain underutilized in biology due to technical barriers such as coding requirements and reliance on cloud infrastructure. We present EZ-FRCNN, a locally hosted, user-friendly package that enables the accessible and scalable application of object detection to biological datasets. Through graphical and scriptable interfaces, users can annotate data, train models, and perform inference entirely offline. We demonstrate its utility in detecting cell phenotypes for large-scale screening, enabling the first label-free tracking of grinder motion in freely moving C. elegans to quantify feeding dynamics, and identifying animals in naturalistic environments for ecological field studies. These once-infeasible analyses now enable rapid screening of cell therapies, investigation of internal state-behavior coupling without immobilization or genetic modification, and efficient wildlife tracking with minimal computational cost. Together, these examples demonstrate how accessible tools like EZ-FRCNN can drive new biological discoveries in both laboratory and field environments.

Whole Slide Images (WSIs) are gigapixel, high-resolution digital scans of microscope slides, providing detailed tissue profiles for pathological analysis. Due to their gigapixel size and lack of detailed annotations, Multiple Instance Learning (MIL) becomes the primary technique for WSI analysis. However, current MIL methods for WSIs directly use embeddings extracted by a pretrained vision encoder, which are not task-specific and often exhibit high variability. To address this, we introduce a novel method, VQ-MIL, which maps the embeddings to a discrete space using weakly supervised vector quantization to refine the embeddings and reduce the variability. Additionally, the discrete embeddings from our methods provides clearer visualizations compared to other methods. Our experiments show that VQ-MIL achieves state-of-the-art classification results on two benchmark datasets. The source code is available at https://github.com/aCoalBall/VQMIL.

In the era of high-throughput biology, molecular phenotypes have proven effective in predicting disease states and future trajectories. Transcriptomics, in particular, has enabled the dissection of complex diseases with heterogeneous genetic and environmental aetiology, both aiding diagnosis and augmenting treatment. As improving technology has led to measurements of gene expression at increasing granularity, it has become progressively feasible to resolve disease traits that present locally or with spatial heterogeneity. Principal among these are cancers in which tumour gene expression, while itself heterogenous, exhibits a distinct signature from that of the surrounding tissue. Identifying this signature through molecular phenotyping facilitates specific cancer diagnosis and treatment. Here, we introduce AGP-Net, a multi-modal foundation framework capable of predicting gene expression from histopathology images. Rather than produce an aggregated estimate of expression for each gene, AGP-Net disaggregates images into spots and attempts to resolve the variation in gene expression across them, thereby providing coarse spatial transcriptomic predictions across the tumour slice and surrounding region. The challenge in doing so is due to data sparsity relative to the dimensionality of the problem: the number of genes and their contextual heterogeneity within and between tissue and cancer types makes it difficult to train a model on the limited data available. The innovation of AGP-Net lies in borrowing strength across similar genes as defined by their textual language descriptions. Our AGP-Net supports datasets with varying gene coverage and facilitates the prediction of gene expression for previously unseen genes based on their textual descriptions. Trained on millions of spots from diverse dataset sources, AGP-Net establishes state-of-the-art performance in zero-shot spatial gene expression prediction, demonstrating its adaptability to generalize across novel scenarios.

Advanced microscopy is a cornerstone of modern science, yet its potential is constrained by a reliance on manual operation, which struggles with the complexity and reproducibility of long-term experiments protocols. While scripting offers a degree of automation, it lacks the generalizability to adapt to new samples or dynamic biological events. A fundamental challenge is the absence of an intelligent system that can interpret high-level scientific intent and ground it in the physical actions of the microscope. To bridge this gap, we introduce an Embodied Intelligent Microscope System (EIMS) with hierarchical reasoning structure that reimagines the microscope autonomous control. This system leverages the advanced reasoning of large models to interpret complex user commands and decompose them into actionable steps. To solve the critical grounding problem, we constrain the models output to the feasible action space, effectively serving as the models "hands and eyes" in the physical world. We demonstrate that our system achieves zero-shot generalization on complex, multi-step protocols and successfully automates challenging biological missions requiring expert-level judgment, such as capturing the sparse spatiotemporal events of cell mitosis and locating scatteredly distributed organoids. This work establishes a new paradigm for scientific instrumentation--fusing high-level intent understanding with grounded, dynamic execution--and provides a generalizable framework for deploying embodied intelligence to accelerate autonomous scientific discovery.

Artificial vision systems hold transformative potential for biomedical imaging, diagnostics, and translational research by emulating and extending the capabilities of biological eyes. However, current techniques often face intrinsic trade-offs between spatial resolution, field of view, and depth perception, particularly in compact, biologically relevant settings. Here, we introduce FOLIC, a foveated light-field compound imaging system, which integrates compound-eye-inspired wide angular coverage and chambered-eye-inspired spatial acuity within a unified multi-aperture concave architecture. FOLIC naturally generates peripheral, blend, and foveated zones from a single capture, enabling seamless, depth-extended, multiscale visualization from wide-field context down to single-cell lateral resolution. We validate FOLIC across diverse fluorescent and non-fluorescent specimens, including cellular phantoms, tissue sections, and small organisms, demonstrating its versatility and scalability for biomedical research and related translational applications. We anticipate FOLIC to offer a biologically informed design blueprint for future artificial vision systems. TeaserA bioinspired system unifies compound and chambered eye principles to achieve wide-field volumetric microscopy.

The lack of annotated publicly available medical images is a major barrier for innovations. At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter. Here we harness these crowd platforms to curate OpenPath, a large dataset of 208,414 pathology images paired with natural language descriptions. This is the largest public dataset for pathology images annotated with natural text. We demonstrate the value of this resource by developing PLIP, a multimodal AI with both image and text understanding, which is trained on OpenPath. PLIP achieves state-of-the-art zero-shot and transfer learning performances for classifying new pathology images across diverse tasks. Moreover, PLIP enables users to retrieve similar cases by either image or natural language search, greatly facilitating knowledge sharing. Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to advance biomedical AI.